The present invention relates to interpolation methods used in digital video signal processing, in particular in frame rate conversion (FRC) applications.
In video processing products using FRC, an input stream of video frames is received with a certain time sampling rate. The FRC process converts such input stream of “original frames” into an output stream having a different time sampling rate, including “interpolated frames” with intermediate time positions, i.e. falling between time positions of the original frames.
Every pixel of the interpolated frames is typically computed as a combination of pixels from the original frames, following a motion vector or direction of invariance determined by analyzing the input stream. More generally, the combination of pixels can make use of one or more interpolation vectors identified by analyzing the evolution of the scene in the video sequence.
Several combination modes can be provided for the interpolation and selected depending on a context determined when analyzing the input stream. In particular, a “fallback mode” for combining the original pixels is sometimes available in the FRC process, which is different from the general operation mode. The fallback mode discards most of the information about the scene (such as object motions, for instance) and performs the interpolation by simply blending pixel values of original frames at the same location as the computed pixel. This amounts to doing an interpolation along a zero motion.
The fallback mode is often used on scenes or parts of scenes which are too complex for the general operation mode to interpolate correctly. In other words, it is used where the FRC engine decides that it is better to leave motion blur or judder instead of introducing unnatural artifacts that will catch the viewer's eye.
Switching between the general operation mode and the fallback mode is very noticeable because it often affects large portions of the image. Although the actual difference between the results of the two modes may be small, the fact of switching can be easily perceived by the final viewer as a sudden jump in the output video sequence. If such a switching happens often and/or for large portions of the image, it can be easily pointed out by the final user as an important artifact of the video processing product using the FRC process.
There is thus a need for interpolation methods with a reduced impact of jumping artifacts.
A video interpolation method is proposed, comprising: analyzing evolution of a scene represented in a video sequence of input frames; and computing output pixels of an output frame having a time position intermediate between time positions of the input frames, by combining respective input pixels of the input frames. At least three interpolation modes are provided for computing the output pixels. The interpolation modes include:                a first mode in which the combined input pixels for an output pixel are selected based on a position of the output pixel in the output frame and a default interpolation vector;        a second mode in which the combined input pixels are selected based on the output pixel position and at least one first interpolation vector determined for the output pixel when analyzing scene evolution; and        a third mode in which the combined input pixels are selected based on the output pixel position and at least one second interpolation vector that is intermediate between the default interpolation vector and a respective first interpolation vector determined for the output pixel when analyzing scene evolution.        
The second mode is a general operation mode based on motion detection or another kind of scene evolution analysis performed in the analysis step, while the first mode is a fallback mode used when the results of the scene analysis are not considered reliable. The third mode uses intermediate interpolation vectors to provide smooth transitions in intermediate situations in a way which corresponds naturally to the properties of the human vision. It is available to avoid jumping artifacts when switching between the general mode and the fallback mode.
In the third mode, the amplitude of the interpolation vectors is gradually changed, which is a much better solution than a simple linear blending between pixels values respectively obtained by the general operation mode and the fallback mode. Human vision acts in such a way that as soon as the fallback mode solution becomes “visible” (i.e. has an interpolation weight substantially different from zero in a linear blending method), it is perceived as a separate scene despite its relative transparency. Thus, blending pixels values alters the smooth transition effect and fails to provide the expected visual result.
The third interpolation mode is typically selected for an output pixel position in a transition phase between use of the first mode and use of the second mode or vice versa for computing output pixels at an output pixel position in successive output frames. The third mode can also be used when the reliability of the scene analysis is degraded but not to the point of switching to the fallback mode. Different ways of handling the transition phases can be implemented.
In an embodiment, a transition degree is determined for an output pixel, and the interpolation mode is selected for said output pixel based on the transition degree. In such a case, when the third interpolation mode is selected for the output pixel, the second interpolation vector may be determined as a function of the transition degree.
In particular, the transition degree can be 0 for the first interpolation mode, 1 for the second interpolation mode and a number between 0 and 1 for the third interpolation mode. For example, when the third interpolation mode is selected for the output pixel, each second interpolation vector may be determined as (vx, vy)=(vxD,vyD)+β·[(v0x, v0y)−(vxD,vyD)], where (vxD,vyD) is the default interpolation vector and (v0x, v0y) is a respective first interpolation vector determined for the output pixel when analyzing scene evolution and β denotes the transition degree (0≦β1). Alternatively, the second interpolation vector can be (vx, vy)=(vxD,vyD)+Min{1,β·vxM/|v0x|,β·vyM/|v0y|}·[(v0x,v0y)−vxD,vyD)] when the third interpolation mode is selected for the output pixel, where v0x and v0y are coordinates of said first interpolation vector along two directions of the frames, vxM and vyM are preset positive parameters, and Min(a,b,c) represents the smallest of three numbers a, b, c.
In an implementation of the interpolation method, determining a transition degree for an output pixel comprises:                obtaining confidence data relating to the first interpolation vectors determined for pixels at the output pixel position in successive output frames; and        analyzing time evolution of the confidence data to obtain the transition degree for the output pixel in a current output frame.        
In a typical non exclusive embodiment, the first interpolation vectors are motion vectors determined for the output pixels by analyzing motion in the step of analyzing evolution of the scene represented in the video sequence. A possibility for the default interpolation vector used in the first and third modes is to take it as a zero motion vector.
Another aspect of the invention relates to a video interpolation apparatus, comprising:                a scene evolution analyzer for analyzing evolution of a scene represented in a video sequence of input frames; and        an interpolator for computing output pixels of an output frame having a time position intermediate between time positions of the input frames, by combining respective input pixels of the input frames. At least three interpolation modes are provided for computing the output pixels, including the above-mentioned first, second and third modes.        
Other features and advantages of the method and apparatus disclosed herein will become apparent from the following description of non-limiting embodiments, with reference to the appended drawings.